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MEMORANDUM 


FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 
AND  TECHNOLOGY 

DIRECTOR,  TEST,  SYSTEMS  ENGINEERING,  AND 


EVALUATION 

BOARD  OF  DIRECTORS  FOR 


TEST  AND  EVALUATION 


SUBJECT;  Audit  Report  on  DoD  Resource  Utilization  Measurement  System 
(Report  No.  95-190) 


We  are  providing  this  report  for  your  review  and  comments.  We  considered 
management  comments  on  the  draft  of  this  report  in  preparing  the  final  report.  As  a 
result  of  management  comments,  we  redirect^  one  recommendation  to  the  Board  of 
Directors  for  Test  and  Evaluation  that  requires  the  establishment  of  Resource 
Utilization  Measurement  System  as  an  internal  control  assessable  unit  and  deleted 
another  recommendation  that  requires  the  reporting  the  lack  of  establishing  Resource 
Utilization  Measurement  System  in  the  Annual  Statement  of  Assurance. 

DoD  Directive  7650.3  requires  that  all  recommendations  be  promptly  resolved. 
Therefore,  the  Board  of  Directors  is  requested  to  provide  final  comments  on  the 
recommendations  by  July  9,  1995. 

The  courtesies  extended  to  the  audit  staff  are  appreciated.  If  you  have  questions 
on  this  audit,  please  contact  Mr.  Raymond  Spencer,  Program  Director,  at  (703)  604- 
9071  (DSN  664-9071)  or  Mr.  Roger  Florence,  Project  Manager,  at  (703)  604-9067 
(DSN  664-9067).  Appendix  G  lists  the  distribution  of  this  report.  The  audit  team 
members  are  listed  inside  the  back  cover. 

Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  95-190  May  9, 1995 

(Project  No.  4AB-5019,02) 

DoD  RESOURCE  UTILIZATION  MEASUREMENT  SYSTEM 
EXECUTIVE  SUMMARY 


Introduction.  In  response  to  congressional  direction,  DoD  proposed  estabhshmg  a 
Resource  Utilization  Measurement  System  (RUMS)  to  report  standardized  workload 
information  that  would  assess  Major  Range  and  Test  Facility  Base  (test  r^ge)  capacity 
and  use.  The  Office  of  the  Secretary  of  Defense  reported  to  Conpess  that 
implementation  of  RUMS  at  all  test  ranges  would  be  accomplished  by  the  end  of 
FY  1994  if  the  system  was  feasible  and  useful. 

Objectives.  The  audit’s  overall  objectives  were  to  evaluate  the  effectiveness  of  the 
implementation  of  RUMS  and  associated  internal  controls. 

Audit  Results.  The  DoD  compliance  with  congressional  direction  to  implement  a 
utilization  reporting  system  for  test  ranges  has  been  tmduly  slow.  The  RUMS  data 
collection  and  reporting  methodologies  were  inconsistent;  outdated  RUMS  dennitions, 
resource  categories,  and  measurement  parameters  were  used  despite  the  issuance  of 
revisions;  and  the  development  effort  had  not  been  effectively  monitored.  As  a  result, 
the  DoD  cannot  use  this  management  tool  to  assist  in  making  accurate  decisions  on 
expansions,  consolidations,  and  improvements  at  the  test  ranges  based  on  as^ssments 
of  capacity  and  use.  A  material  internal  control  weakness  existed  because  DoD  had  not 
adequately  monitored  the  implementation  of  RUMS  and  the  adequacy  of  information 
provided  The  RUMS  will  be  an  integral  part  of  the  internal  control  structure  for  test 
r^ge  management.  Parti  of  this  report  discusses  the  internal  controls  assessed  and 
Part  n  discusses  the  details  on  the  internal  control  weakness. 

Summary  of  Recommendations.  We  recommend^  that  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  direct  the  implementation  of  RUMS  the 
19  test  ranges  by  the  end  of  FY  1995.  We  recommended  that  the  Director,  Test, 
Systems  Engineering,  and  Evaluation,  take  an  active  role  in  the  establishment  of 
RUMS.  We  also  recommended  that  the  Board  of  Directors  for  Test  and  p^u^on 
designate  the  establishment  of  RUMS  as  an  assessable  unit  in  accordance  with  the  DoD 
Internal  Management  Control  Program. 

Management  Comments.  Management  comments  were  received  from  the  Director, 
Test,  Systems  Engineering,  and  Evaluation.  His  comments  included  responses  from 
the  Military  Departments.  The  Director  nonconcurred  with  the  recommendation  to  take 
an  active  role  in  RUMS  development  and  the  responses  from  the  MUitary  Department 
partially  concurred  with  our  recommendation.  The  Director  stated  that  he  already 
actively  monitors  the  RUMS  development  and  believes  that  oversight  provided  by  the 
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Board  of  Directors,  the  Board  of  Operating  Directors,  and  other  Reliance  boards  is 
appropriate.  The  Director  and  the  Military  Departments  partially  concurred  with 
designating  RUMS  as  an  internal  control  assessable  unit.  The  Director  stated  that  he 
would  recommend  that  the  Board  of  Operating  Directors  designate  RUMS  as  an 
assessable  unit.  The  Director  and  the  Military  Departments  nonconcurred  with 
reporting  the  lack  of  RUMS  development  as  a  material  internal  control  weakness 
because  they  stated  that  RUMS  was  on  track  with  its  revised  schedule.  The  Director 
and  the  Military  Departments  agreed  with  implementing  RUMS  by  the  end  of 
FY  1995;  however,  the  Director's  comments  stated  that  the  Board  of  Operating 
Directors  should  have  this  responsibility. 

Audit  Response.  Although  the  management  comments  either  nonconcurred  or 
partially  concurred  with  the  audit  recommendations,  planned  management  actions  were 
responsive  to  the  recommendations'  intent.  The  comments  indicated  that  RUMS  would 
be  implemented  by  the  end  of  FY  1995  and  the  Director,  Test,  Systems  Engineering, 
and  Evaluation,  agreed  that  RUMS  should  be  designed  as  an  internal  control  assessable 
unit.  The  basis  of  the  nonconcurrences  was  either  when  the  action  would  be  taken  or 
who  should  be  responsible  for  taking  the  action.  As  a  result  of  management  comments, 
we  redirected  one  recommendation  to  the  Board  of  Directors  for  Test  and  Evaluation 
concerning  establishing  RUMS  as  an  assessable  unit  and  deleted  another 
recommendation  to  report  a  material  internal  control  weakness.  If  RUMS  remains 
nonoperational  as  of  September  30,  1995,  however,  the  Inspector  General  will 
recommend  to  the  Secret^  of  Defense  that  a  material  weakness  be  reported  in  the 
Annual  Assurance  Statement. 

We  request  that  the  Chairman  of  the  Board  of  Directors  for  Test  and  Evaluation 
provide  comments  to  the  redirected  recommendation  by  July  9,  1995. 
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This  report  was  prepared  by  the  Acquisition  Management  Directorate,  Office  of  the  Assistant 
Inspector  General  for  Auditing,  Department  of  Defense. 


Part  I  -  Introduction 


Background 


The  Military  Departments  operate  and  maintain  the  19  Major  Range  and  Test 
Facility  Bases  (test  ranges);  the  Office  of  the  Secretary  of  Defense  (OSD)  is 
responsible  for  providing  policy  and  oversight  of  the  ranges.  The  six  Army, 
six  Navy,  and  seven  Air  Force  test  ranges  support  DoD  Components,  U.S. 
Government  Agencies,  foreign  governments,  and  private  organizations.  Test 
ranges  comprise  large  land  and  sea  areas  and  air  space  divided  into  various  test 
sites.  Generic  equipment  and  instrumentation  such  as  tracking  radars  and  threat 
emitters  are  located  throughout  the  test  ranges,  while  individu^  test  sites  contain 
equipment  and  instrumentation  required  for  specific  types  of  tests.  Test  ranges 
are  used  for  a  variety  of  test  purposes  that  include  aircraft,  bombs,  missiles, 
tanks,  vehicles,  environmental  effects  on  weapon  systems,  and  underwater 
testing  of  munitions  and  submarines.  The  19  test  ranges  represent  a  DoD 
investment  of  $20  billion  to  $30  billion;  have  an  annual  operating  budget  of 
approximately  $5  billion;  and  employ  more  than  54,000  military,  civilian,  and 
contractor  personnel. 


Objectives 


The  audit's  overall  objectives  were  to  evaluate  the  effectiveness  of  the 
implementation  of  a  Resource  Utilization  Measurement  System  (RUMS)  and 
associated  internal  controls. 


Scope  and  Methodology 


This  economy  and  efficiency  audit  was  made  from  March  through 
December  1994  in  accordance  with  auditing  standards  issued  by  the  Comptroller 
General  of  the  United  States  as  implemented  by  the  Inspector  General,  DoD, 
and  accordingly  included  such  tests  of  intemd  controls  as  were  considered 
necessary.  This  audit  included  one  Army,  one  Navy,  and  two  Air  Force  test 
ranges  that  were  selected  to  participate  in  the  development  and  field  trial  of  the 
RUMS.  (The  Naval  Air  Warfare  Center  -  Weapons  Division  was  not  included 
in  the  audit  due  to  other  audit  effort  at  that  activity).  We  analyzed  the  RUMS 
reports  from  October  1992  through  September  1994  to  determine  whether 
workload  and  utilization  data  were  reported  in  accordance  with  applicable 
guidance.  However,  we  did  not  perform  audit  tests  to  establish  the  validity  of 
the  computer-processed  workload  and  utilization  data.  Specifically,  we  did  not 
examine  the  original  utilization  records  to  determine  whether  the  information 
entered  into  the  computer  systems  was  complete  and  accurate.  We  reviewed  the 
mission,  function,  and  management  responsibilities  of  the  Tri-Service  RUMS 
Committee  (the  Tri-Service  Committee)  that  was  responsible  for  completing 


2 


Introduction 


RUMS  development.  We  reviewed  Tri-Service  Committee  minutes  of  meetings 
to  determine  whether  issues  raised  were  appropriately  resolved.  Organizations 
visited  or  contacted  during  the  audit  are  list^  in  Appendix  F. 


Internal  Controls 


We  evaluated  internal  controls  related  to  the  review  process  and  the  adequacy  of 
information  in  support  of  RUMS  development.  We  reviewed  OSD's  and  the 
field  trial  test  ranges'  implementation  of  the  DoD  Internal  Management  Control 
Program  as  it  pertained  to  the  RUMS  development.  We  identified  a  material 
internal  control  weakness  as  defined  in  DoD  Directive  5010.38,  "Internal 
Management  Control  Program,"  April  14,  1987.  Specifically,  OSD  and  the 
Military  Departments  failed  to  implement  the  RUMS,  which  is  an  essential 
element  in  the  internal  control  structure  for  range  and  test  facility  management. 
Also,  neither  OSD  nor  the  field  trial  test  ranges  identified  RUMS  as  an 
assessable  unit  or  had  identified  internal  control  weaknesses  attributable  to  the 
delays  in  the  RUMS  development  effort.  Recommendation  3  will  correct  these 
weaknesses.  If  RUMS  remains  nonoperational  as  of  September  30,  1995,  it 
should  be  addressed  as  a  material  internal  control  problem  in  the  DoD  Annual 
Assurance  Statement  for  FY  1995. 

The  benefits  of  implementing  the  recommendations  are  nonmonetary. 
However,  implementation  of  Recommendation  3  will  result  in  accurate  ^d 
timely  RUMS  reporting  and  provide  a  management  tool  for  resource  utilization 
comparisons.  A  copy  of  the  final  report  will  be  provided  to  senior  officials  in 
DoD  responsible  for  internal  controls. 


Prior  Audit 


The  General  Accounting  Office  (GAO)  issued  audit  report  GAO/NSIAD  90-91 
(OSD  Case  No.  8451),  "Test  and  Evaluation:  A  Proposed  Framework  for 
Measuring  the  Use  of  Test  Facilities,"  August  8,  1990.  GAO  examined  the 
capacity  and  use  of  the  test  ranges  and  assessed  whether  adequate  measures  of 
data  collection  and  reporting  were  established.  GAO  proposed  a  framework  that 
DoD  could  use  to  measure  capacity  and  use  at  test  ranges.  GAO  made  no 
recommendations.  GAO  discussed  the  report's  conclusions  with  OSD  officials 
responsible  for  managing  the  test  ranges;  the  officials  generally  agreed  that  the 
approach  had  potentid  and  could  be  easily  implemented. 
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Part  II  -  Finding  and  Recommendations 


DoD  Resource  Utilization  Measurement 
System 

DoD  compliance  with  congressional  direction  to  implement  a  utilization 
measurement  system  at  test  ranges  has  been  unduly  slow.  OSD  and  the 
Military  Departments  had  not  been  effectively  monitoring  the  Resource 
Utilization  Measurement  System  (RUMS)  implementation.  Problems 
with  implementation  of  RUMS  at  the  trial  ranges  included  inconsistent 
data  collection  methodologies  and  the  use  of  outdated  RUMS  definitions, 
resource  categories,  and  measurement  parameters.  As  a  result,  the  OSD 
and  the  Military  Departments  cannot  yet  use  this  management  tool  to 
assist  in  making  accurate  decisions  on  consolidation,  expansion,  and 
improvements  at  the  test  ranges  based  on  credible  assessments  of 
capacity  and  use.  However,  implementation  is  now  planned  by  the  end 
of  FY  1995. 


Background 


In  1988,  a  DoD  Range  Commanders  Council  established  a  working  group  to 
develop  a  utilization  measurement  system  to  identify  each  test  range's  capacity 
and  measure  usage.  However,  OSD  officials  responsible  for  test  range 
management  did  not  adopt  the  system  because  they  believed  that  implementing 
the  system  would  be  labor  intensive  and  costly. 

Later  congressional  concern  over  DoD's  inability  to  compare  workload  and 
utilization  at  test  ranges  prompted  the  House  Appropriations  Committee,  during 
the  appropriations  process  for  FY  1991,  to  direct  the  DoD  to  implement  a 
framework  for  measuring  usage  at  the  test  ranges.  Specifically,  a  House  of 
Representatives,  Committee  on  Appropriations  Report  101-822  directed  that 
DoD  institute  the  framework  suggested  in  the  GAO  1990  audit  report,  or  an 
equivalent  system,  and  report  the  results  of  its  efforts  to  the  House 
Appropriations  Committee  by  June  1991. 

In  responding  to  the  House  Appropriations  Committee  in  July  1991,  the 
Director  of  Defense  Research  and  Engineering  proposed  a  phased  development 
schedule  for  implementing  a  utilization  measurement  system.  The  phased 
approach  proposed  that  during  FY  1991,  DoD  would  design  RUMS;  during 
FY  1992,  field  trials  would  be  conducted  at  five  test  ranges  ,  during  FY  1993, 

^The  five  field  trial  test  ranges  selected  to  participate  in  the  RUMS  development 
were  the  Army's  White  Sands  Missile  Range,  New  Mexico;  the  Naval  Air 
Warfare  Center  -  Aircraft  Division,  Patuxent  River,  M^land;  the  Naval  Air 
Warfare  Center  -  Weapons  Division,  China  Lake,  California;  the  Air  Force 
Flight  Test  Center,  Edwards  Air  Force  Base,  California;  and  the  Air  Force 
Development  Test  Center,  EgUn  Air  Force  Base,  Florida.  These  test  ranges 
were  selected  because  they  were  representative  of  the  scope  and  breadth  of 
missions  performed  by  the  ranges. 
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DoD  would  assess  the  feasibility  and  usefulness  of  RUMS;  and  DoD  would 
implement  RUMS  at  all  19  test  ranges  by  the  end  of  FY  1994  if  the  system  was 
feasible  and  useful. 

The  Senate  Appropriations  Committee  also  expressed  concern  over  DoD's 
inability  to  compare  workload  and  utilization  at  test  ranges.  In  a  Senate 
Appropriations  Committee  Report  102-154  concerning  the  DoD  Appropriations 
Bill  for  FY  1992,  the  Committee  directed  DoD  to  develop  a  standard 
measurement  system  that  would  define  the  use  of  each  Military  Department's 
test  facilities  to  support  requested  funding  for  FY  1993.  As  a  result,  in 
May  1993,  the  Board  of  Operating  Directors^  (formerly  the  Joint  Commanders 
Group  for  Test  and  Evaluation)  directed  that  a  Tri-Service  RUMS  Committee 
(the  Tri-Service  Committee)  be  formed  with  two  representatives  from  each 
Military  Department  to  complete  the  RUMS  development.  The  Tri-Service 
Committee  was  tasked  to  finalize  the  RUMS  development. 

The  Tri-Service  Committee  developed  definitions  and  measurement  parameters 
that  enabled  the  test  ranges  to  perform  trend  analyses  on  the  use  of  their  test 
capabilities.  Appendix  A  provides  the  description  of  RUMS  definitions  and 
parameters.  Further,  the  Military  Department  representative  from  the  test 
ranges  developed  20  functional  categories  that  represent  the  majority  of  test 
ranges'  facilities  and  capabilities.  The  20  functional  categories  have  examples 
of  test  facilities  that  should  be  in  each  category  or  an  expanded  description  of 
the  category.  Appendix  B  provides  a  list  and  description  of  the  20  functional 
categories. 

DoD  Directive  3200.11,  "Major  Range  and  Test  Facility  Base,"  September  29, 
1980,  assigns  responsibilities;  functions;  oversight  of  test  ranges;  and  budgetary 
decisions  on  range  expansion,  consolidation,  and  improvements  to  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  (formerly  the  Under 
Secretary  of  Defense  for  Research  and  Engineering).  The  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  designated  the  Director,  Test,  Systems 
Engineering,  and  Evaluation  (formerly  the  Director  for  Test  and  Evaluation),  as 
the  official  responsible  for  establishing  test  range  policy.  The  Secretaries  of  the 
Military  Departments,  under  policy  guidance  and  oversight  of  the  Director  for 
Test  Systems  Engineering,  and  Evaluation,  are  responsible  for  management  of 
test  ranges  under  their  cognizance.  The  Military  Departments  established  the 
Board  of  Directors  to  oversee  test  and  evaluation  infirastructure  investment, 
infrastructure  consolidation,  standards,  and  policy.  The  Board  of  Operating 
Directors  was  established  to  implement  the  Board  of  Directors'  decisions. 

DoD  Directive  5010.38,  "Internal  Management  Control  Program,"  April  14, 
1987,  establishes  policies  and  procedures  for  DoD  organizations  for  internal 
management  control.  These  policies  and  procedures  require  Federal  managers 


^The  Board  of  Operating  Directors  is  a  Tri-Service  functional  group  under  the 
authority  of  the  Board  of  Directors. 
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to  implement  a  comprehensive  system  of  internal  controls  and  identify 
assessable  units^  that  will  provide  reasonable  assurance  that  assets  are  used 
properly  and  programs  are  effectively  and  efficiently  managed. 


Definitions  and  Measurement  Parameters 


In  developing  an  approach  for  RUMS,  the  Tri-Service  Committee  developed 
definitions  and  measurement  parameters.  The  pu^ose  of  the  definitions  and 
measurement  parameters  was  to  promote  standardization  in  reporting  of  trial 
range  data.  The  parameters  measure  capacity  and  use  efficiency  and  were 
intended  to  assist  in  identifying  under-  or  over-utilized  test  facilities. 

For  test  facilities  that  were  reported,  trial  ranges  did  hot  report  all  measurement 
parameters.  Appendix  C  shows  our  analysis  of  RUMS  and  identifies  the 
unreported  parameters  at  the  four  trial  ranges  included  in  our  review.  For 
example,  the  Army's  White  Sands  Missile  Range  (WSMR)  reported  six  test 
fecilities  in  RUMS  and  reported  the  parameters  incorrectly.  Incorrect 
parameters  were  reported  because  the  WSMR  used  outdated  RUMS  definitions 
and  parameters.  The  outdated  information  was  submitted  to  the  Army  RUMS 
Tri-Service  Committee  representative  and  its  validity  was  not  questioned. 
WSMR  officials  told  us  that  while  they  had  expressed  their  concerns  regarding 
the  reporting  of  RUMS  data  to  their  headquarters,  the  Army  Test  and 
Evaluation  Command,  they  had  received  no  revised  guidance  or  response  to 
their  concerns. 

The  Air  Force  Flight  Test  Center  at  Edward  Air  Force  Base  was  not  reporting 
"unconstrained  capacity"  in  accordance  with  applicable  guidelines. 
Unconstrained  capacity  measures  the  maximum  time  a  resource  could  be  used 
without  personnel  limitations.  The  Air  Force  Flight  Test  Center  RUMS 
reported  the  anechoic  chamber  and  the  Integrated  Facility  for  Avionics  Systems 
Testing's  unconstrained  capacity  was  16  hours  per  day  for  a  5 -day  week. 
According  to  the  definition  of  unconstrained  capacity,  this  facility  should  be 
reported  at  16  hours  per  day  for  7  days  a  week.  Officials  at  the  Air  Force 
Flight  Test  agreed  that  both  test  facilities  could  be  used  16  hours  per  day  for  a 
7-day  week.  These  test  facilities  are  usually  limited  to  16  hours  of  operation 
due  to  required  maintenance  or  repair  and  test  setup  and  dismantle.  As  a  result, 
these  facilities'  capacities  were  underreported  by  32  hours  per  week. 


^An  assessable  unit  is  defined  as  any  organization  or  function  capable  of  being 
evaluated  by  internal  management  control  review. 


8 


DoD  Resource  Utilization  Measurement  System 


Methodology  and  Techniques 


We  reviewed  methodologies  and  techniques  for  gathering  RUMS  data  at  four  of 
the  five  field  trial  test  ranges  (the  trial  ranges)  and  found  that  some  test  facilities 
and  capabilities  were  not  reported  and  reporting  was  inconsistent. 

Unreported  Test  Facilities.  Trial  ranges  were  also  not  complying  with  the  Tri- 
Service  Committee's  guidance,  resulting  in  test  facilities  not  being  reported. 
For  example,  the  Naval  Air  Warfare  Center  -  Aircraft  Division  did  not  report 
10  out  of  20  test  facilities,  all  of  which  were  identified  under  the  functional 
category  detail  description  as  reportable  facilities.  Also,  the  Air  Force 
Development  Test  Center  did  not  report  7  out  of  14  test  facilities;  all  of  these 
facilities  were  identified  in  the  detail  description  for  functional  categories.  All 
test  fecilities  at  the  WSMR  and  the  Air  Force  Flight  Test  Center  were  reported. 
Appendix  C  lists  the  reported  and  unreported  facilities  by  trial  range. 

Inconsistent  Reporting.  The  reporting  of  RUMS  data  among  the  trial  ranges 
was  inconsistent  for  similar  facilities.  Some  trial  ranges  collected  RUMS  data 
for  functional  categories  while  other  trial  ranges  did  not.  For  example,  the 
Naval  Air  Warfare  Center  -  Aircraft  Division,  Patuxent  River,  Maryland, 
reported  utilization  data  for  its  over-water  range,  the  Chesapeake  Test  Range. 
However,  the  Air  Force  Development  Test  Center,  Eglin  Air  Force  Base, 
Florida,  had  not  reported  utilization  data  for  its  over-water  range,  the  Gulf 
Range  Water  Test  Area.  The  description  of  the  functional  category  for  Test 
Range  Complex  states  that  "major  air  and  ground  space"  is  a  reportable  test 
capability.  The  Air  Force  Development  Test  Center  personnel  did  not  report 
the  86,500  square  mile  Gulf  Range  because  they  considered  it  a  minor  test 
capability.  However,  the  Tri-Service  Committee  identified  the  over-water  range 
as  a  reportable  capability. 

Also,  reporting  of  facilities  that  had  mission  control  systems  for  in-flight 
operations  was  inconsistent.  The  mission  control  systems  provide  in-flight 
safety,  data  collection  processing  and  display,  operation  control  capabilities,  and 
range  communication.  The  Air  Force  Flight  Test  Center  and  the  Naval  Air 
Warfare  Center  -  Aircraft  Division  reported  their  facilities  for  mission  control  as 
required  in  the  functional  categories.  However,  the  Air  Force  Development 
Test  Center  did  not  report  utilization  for  the  same  type  of  facility.  The  Air 
Force  Development  Test  Center  did  not  report  this  system  because  it  considered 
the  facility  to  be  minor.  The  detail  description  of  the  functional  categories 
specifies  that  a  mission  control  facility  is  a  reportable  capability. 


Monitoring  Implementation  Effort 


We  met  with  Tri-Service  Committee  representatives  to  discuss  the  overall 
RUMS  development.  Although  the  Tri-Service  Committee  identified  and 
documented  problems  with  tihe  RUMS  development  and  issued  revised 
guidance,  little  evidence  showed  that  program  execution  was  monitored.  The 
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Tri-Service  Committee  was  not  aware  of  reporting  inconsistencies  that  we 
identified  at  the  trial  ranges.  These  conditions  resulted  in  incorrect  resource 
utilization  data  and  precluded  accurate  comparison  of  utilization  data  between 
the  test  ranges. 

A  Navy  official,  who  was  Chairman  of  the  Tri-Service  Committee  during  the 
RUMS  development,  stated  that  it  was  not  the  Tri-Service  Committee's 
intention  that  il  test  capabilities  be  reported.  He  stated  that  the  Tri-Service 
Committee  decided  that  reporting  could  be  limited  to  80  percent  of  reportable 
test  capabilities.  However,  the  Navy  official  was  unable  to  provide 
documentation  supporting  the  Tri-Service  Committee's  decision  to  limit 
reportable  test  capabilities.  We  determined  that  the  Naval  Air  Warfare  Center  - 
Aircraft  Division  was  only  reporting  50  percent  of  the  testing  capabilities. 

The  Tri-Service  Committee  official  at  the  Air  Force  Development  Test  Center 
decided  that  only  85  percent  of  its  test  capabilities  would  be  reported.  The  Tri- 
Service  Committee  official  stated  that  the  test  capabilities  that  were  not  reported 
were  considered  "minor"  capabilities.  However,  the  Air  Force  Tri-Service 
Committee  official  was  unable  to  provide  documentation  that  provided 
classification  of  "major"  or  "minor"  capabilities. 

Our  examination  of  Tri-Service  Committee  documentation  did  not  show  that 
facility  reporting  could  be  limited  to  80  or  85  percent  of  the  20  functional 
categories.  Reporting  less  than  100  percent  of  test  capabilities  and  usage  would 
prohibit  OSD  and  the  Military  Departments'  management  from  comparing  all 
testing  resources. 


Use  of  Utilization  Data 


The  Defense  Base  Realignment  and  Closure  Commission  (the  BRAC)  is 
responsible  for  ensuring  a  timely,  independent,  and  fair  process  for  realigning 
and  closing  DoD  installations.  In  developing  Realignment  and  Closure 
recommendations,  the  Test  and  Evaluation  Joint  Cross-Service  Working  Group 
(a  BRAC-related  sub-group)  used  utilization  information  that  was  developed 
independently  of  RUMS.  TTie  sub-group  did  not  use  RUMS  data  because  it 
concluded  that  the  RUMS  data  were  not  reliable.  Inspector  General,  DoD, 
audit  report  No.  95-134,  "Inspector  General,  DoD,  Concurrence  With  Joint 
Cross-Service  Group  Recommendation  to  Change  Methodology  Used  to 
Determine  Excess  Capacity,"  Februai^  27,  1995,  identifies  the  difficulty  the 
working  group  experienced  in  attempting  to  use  RUMS  data.  To  identify  range 
capacity,  the  working  group  used  a  "Historical  Peak  Workload  Method."  To 
ensure  that  responses  to  requests  for  clarification  by  the  working  group  were 
accurate,  the  Deputy  Assistant  Secretary  of  Defense  for  Economic  Reinvestment 
requested  that  the  Inspector  General  review  supporting  documentation  of 
information  provided  to  this  BRAC  working  group.  In  response  to  this  request, 
the  Inspector  General  had  audit  teams  evaluate  documentation  at  five  test 
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ranges.  Had  the  Military  Departments  implemented  an  effective  RUMS  at  all 
19  test  ranges  as  planned,  an  alternative  measurement  system  would  not  have 
had  to  be  used  and  additional  validation  efforts  would  have  been  unnecessary. 


Conclusion 


Although  DoD  reported  to  Congress  in  July  1991  that  the  Military  Departments 
would  implement  RUMS,  if  feasible,  at  all  test  ranges  in  FY  1994,  the  Military 
Departments  did  not  meet  this  milestone.  The  development  of  RUMS  for  the 
test  ranges  has  taken  too  much  time.  The  Military  Departments'  reluctance  to 
implement  an  effective  measurement  system  has  persisted  since  1988.  The 
Military  Departments  may  never  achieve  a  standardized  RUMS  at  the  test 
ranges  without  the  RUMS  implementation.  Unless  the  Department  of  Defense 
takes  a  strong  position  on  implementing  RUMS  and  establishing  RUMS  as  ^ 
assessable  unit  in  an  Internal  Management  Control  Plan,  continued  delays  in 
RUMS  implementation  will  occur. 


Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  reconunend  that  the  Director,  Test,  Systems  Engineering,  and 
Evaluation,  take  an  active  role  in  monitoring  the  implementation  of  the 
Resource  UtUization  Measurement  System  at  the  Major  Range  and  Test 
Facility  Base  to  ensure  its  accuracy  and  completeness. 

Management  Comments.  The  Director,  Test,  Systems  Engineering,  and 
Evaluation  (the  Director),  nonconcurred  with  the  draft  report  recommendation 
to  take  an  active  role  in  the  RUMS  development.  The  Director's  response 
included  comments  from  the  Military  Departments  and  the  Military 
Departments  partially  concurred  with  draft  report  recommendation. 

The  Director  and  the  Military  Departments’  comments  indicated  that  the 
Director  had  taken  an  active  role  in  RUMS  development  by  receiving  frequent 
briefings.  The  Director  and  the  Military  Departments  believed  that  the  Board  of 
Operating  Directors,  under  the  authority  of  the  Board  of  Directors  for  Test  and 
Evaluation,  should  be  responsible  for  RUMS  implementation.  The  Director  and 
the  Military  Departments'  comments  stated  that  RUMS  will  be  implemented  by 
the  end  of  FY  1995. 

The  Military  Department  comments  also  stated  that  RUMS  will  become  another 
tool  for  local  managers  to  efficiently  manage  their  facilities,  but  not  a  tool  that 
will  compare  one  facility  with  another  for  macro  decisionmaldng.  The  complete 
text  of  management  comments  is  in  Part  IV. 
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Audit  Response.  The  intent  of  the  recommendation  to  the  Director,  Test, 
Systems  Engineering,  and  Evaluation,  was  to  provide  oversight  in  the  RUMS 
development.  We  made  this  recommendation  because  RUMS  has  been  in 
various  stages  of  development  since  1988.  We  believe  that  this  amount  of  time 
was  excessive.  However,  we  consider  management  comments  to  be  responsive 
to  the  intent  of  the  recommendation  because  the  comments  stated  that  RUMS 
would  be  implemented  at  the  test  ranges  by  the  end  of  FY  1995. 

We  agree  that  activity  managers  can  use  RUMS  as  a  management  tool  to 
measure  utilization.  However,  the  RUMS  can  do  far  more  than  provide  only 
local  managers  with  a  system  to  determine  testing  resource  utilization.  We  are 
not  asserting  that  RUMS  data  alone  would  be  sufficient  to  support  future 
BRAC  -  type  processes. 

2.  We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  direct  the  implementation  of  a  Resource  Utilization 
Measurement  System  at  all  test  ranges  by  the  end  of  FY  1995. 

Management  Comments.  The  Director  provided  comments  for  the  Under 
Secretary  of  Defense  for  Acquisition  and  Technology.  The  Director  stated  that 
RUMS  would  be  implement^  at  all  test  ranges  by  the  end  of  FY  1995.  The 
Military  Departments  agreed  with  that  commitment. 

Audit  Response.  Comments  provided  by  the  Director  are  responsive  to  the 
recommendation. 

3.  We  recommend  that  the  Chairman  of  the  Board  of  Directors  for  Test 
and  Evaluation  establish  the  Resource  Utilization  Measurement  System  as 
an  assessable  unit  in  accordance  with  the  DoD  Internal  Management 
Control  Program. 

Management  Comments.  The  Director,  in  response  to  the  draft  report,  stated 
that  he  would  recommend  to  the  Board  of  Operating  Directors  that  RUMS  be 
designated  an  assessable  unit.  As  a  result  of  the  Director's  comments,  we  have 
redirected  this  recommendation  to  the  Board  of  Directors. 

The  Military  Departments  partially  concurred  with  the  draft  report 
recommendation  for  establishing  RUMS  as  an  internal  control  assessable  unit. 

Audit  Response.  We  consider  management  comments  to  be  responsive  to  the 
recommendation.  The  Director  and  the  Military  Departments  believed  that  the 
Board  of  Operating  Directors,  under  the  authority  of  the  Board  of  Directors, 
should  have  the  responsibility  for  ensuring  RUMS  implementation.  In  response 
to  management  comments,  we  redirected  the  recommendation  for  establishing 
RUMS  as  a  assessable  unit  to  the  Board  of  Directors  for  Test  and  Evaluation. 

The  Director  and  the  Military  Departments  provided  additional  comments  to  the 
issues  discussed  in  the  draft  report.  Appendix  D  provides  a  summary  of  the 
comments  and  our  response. 
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Part  III  -  Additional  Information 


Appendix  A.  Resource  Utilization  Measurement 

System  DeHnitions  and  Parameters 


Definitions 


Budgeted  Capacity.  A  measure  of  the  maximum  amount  of  time  a  test 
resource  is  staffed  under  normal  personnel  conditions,  including  the  use  of 
contractor  personnel.  Budgeted  capacity  excludes  overtime. 

Nonavailability.  The  amount  of  time  a  test  resource  is  not  available  for 
customer  use  due  to  external  constraints  (for  example,  test  cancellations, 
maintenance,  and  weather). 

Overtime.  The  amount  of  resource  use  in  excess  of  budgeted  capacity. 

Resource.  A  test  facility  or  capability  (including  air,  land,  and  sea  space)  and 
associated  equipment  needed  to  conduct  a  test  and  evaluation. 

Surge  Capacity.  Unconstrained  capacity  minus  budgeted  capacity. 

Time.  A  measurement  in  clock  hours,  allowing  for  simultaneous  multiple 
testing  at  a  test  facility  or  a  range.  If  user  test  hours  are  defined  in  multiples 
(for  example,  more  than  one  test  hour  per  clock),  then  capacity  must  be  defined 
in  multiple  hours. 

Unconstrained  Capacity.  A  measure  of  the  maximum  amount  of  time  a  test 
resource  could  be  used  with  unconstrained  personnel  conditions. 

User  Time.  A  measure  of  time  a  test  resource  is  paid  for  that  includes  set  up 
and  dismantle  time  if  it  precludes  use  of  the  resource  by  another  customer). 
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Appendix  A.  Resource  Utilization  Measurement  System  Definitions  and 

Parameters 


Parameters 


0  Use  efficiency.  A  measure  of  the  percent  of  time  a  resource  is  available  for 
customer  use. 

Use  Efficiency  =  Budgeted  Capacity  minus  Nonavailability 
Budgeted  Capacity 


o  Utilization.  A  measure  of  resource  use  to  budgeted  capacity  (level  of 
fimding)  plus  overtime. 

Utilization  = _ User  Time  _ 

Budgeted  Capacity  plus  Overtime 


o  Surge  capacity.  A  measure  of  the  maximum  capacity  potential  of  a  facility 
given  an  unconstrained  budget  and  personnel  environment  (without  additional 
infrastructure  investment). 

Surge  Capacity  =  Unconstrained  Capacity  minus  Budgeted  Capacity 


o  Surge.  A  measure  of  the  percentage  of  maximum  capacity  potentially 
available  for  customer  use. 

Surge  =  Unconstrained  Capacity  minus  Budgeted  Capacity 
Unconstrained  Capacity 
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Appendix  B.  Resource  Utilization  Measurement 

System  Functional  Categories 


1 .  Acoustic  Test  Facility  (provides  the  capability  to  test  and  evaluate  the 
effects  of  anti-submarine  warfare  acoustic  sensor  signals), 

2.  Aircraft  Anechoic  Facility  (provides  testing  of  integrated  electronics  and 
weapon  systems  by  stimulating  the  systems  sensors). 

3.  Aircraft  Test  and  Evaluation  Facility  (provides  ground  testing  of  aircraft 
engine  systems  and  components  includes  tests.  Test  may  include  the  effects  of 
heat,  rain,  and  thrust). 

4.  Avionics  Test  Facility  (provides  stimulation,  control,  and  monitoring  of 
individual  avionics  systems  not  possible  when  installed  on  an  aircraft). 

5.  Catapult  and  Arresting  Gear  Facility  (simulates  ship  environment  for 
aircraft  take  off  and  landing). 

6.  Climatic  Environment  Facilities  (provides  testing  capability  under  a 
variety  of  conditions  such  as  extreme  cold  and  high  moisture). 

7.  Dynamic  Environment  Facilities  (facilities  that  provide  shock,  vibration, 
acoustic,  and  seismic  testing). 

8.  Electromagnetic  Radiation  Facilities  (provides  testing  for  susceptibility 
and  vulnerability  of  electronic  components). 

9.  Electronic  Warfare  Test  and  Evaluation  Range  (provides  test  and 
evaluation  of  electronic  combat  devices,  components,  systems,  and  techniques 
against  simulated  hostile  systems). 

10.  Guided  Weapons  Evaluation  Facility  (provides  for  tests  of  precision 
guided  weapons). 

11.  Landing  System  Test  Facility  (provides  testing  for  aircraft  automatic, 
semi-automatic,  and  visual  landing  systems). 

12.  Manned  Flight  Simulator  (provides  flight  test  programs  of  flight 
dynamics  and  aircraft  systems  that  are  used  to  analyze  flying  qualities  and  flight 
control  systems). 

13.  Nuclear  Effects  Test  Facility  (simulates  effects  associated  with  nuclear 
weapon  detonation). 

14.  Propulsion  Test  Facility  (testing  capability  of  rocket  motors  and  missile 
propulsion  systems). 
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15.  Radar  Cross-Section  Range  (provides  precision  radar  measurements  of 
components  in  an  outdoor  environment). 

16.  Safety  Test  Facility  (provides  weapons  safety  tests  and  weapons 
reactions  to  various  hazards  such  as  fires,  bullet,  and  dropping  impacts). 

17.  Shielded  Hangar  (provides  an  electromagnetic  environment  for  electronic 
testing). 

18.  Test  Range  Complex  (includes  air  and  ground  space  areas, 
instrumentation  radars,  range  and  mission  control  facilities,  targets,  and  tracking 
systems). 

19.  Test  Track  (rail  sled  system  to  test  systems  at  subsonic,  supersonic  or 
hypersonic  speeds). 

20.  Warhead  Test  Facility  (provides  fragment  velocity  and  dispersion,  air 
blast  measurements,  and  warhead  effectiveness  against  various  targets). 
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Table  C-1.  Annv  White  Sands  Missile  Ranee.  New  Mexico 


Appendix  C.  Analysis  of  Resource  Utilization 

Measurement  System 
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Table  C-2.  Naval  Air  Warfare  Center  -  Aircraft  Division.  Patuxent  River.  Maryland 
_ Determined  Within  Guidelines? _ 
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Test  facilities  with  these  specific  capabilities  were  identified  as  reportable  under  the  functional  categories. 


Table  C-3,  Air  Force  Development  Test  Center.  Eglin  Air  Force  Base.  Florida 
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12.  Gun  Test  Facility* 

13.  Security  Systems  Test 
Facility  Complex 

14.  Centralized  Control 
Facility 
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Appendix  D.  Analysis  of  Management  Comments 

to  the  Draft  Audit  Report 


The  Director,  Test,  Systems  Engineering,  and  Evaluation  (the  Director)  and  the 
Military  Departments  provided  general  comments  to  issues  discussed  in  the  draft 
report.  Below  is  a  summary  of  the  issues  discussed  in  the  management 
comments  and  the  audit  response.  The  complete  text  of  the  management 
comments  is  in  Part  IV. 

Management  Comments.  The  Director  stated  that  the  Office  of  the  Secretary 
of  Defense  did  not  report  to  Congress  that  RUMS  would  be  implemented  by  the 
end  of  FY  1994.  He  stated  that  in  response  to  a  GAO  report,  the  Director 
stated  that  if  the  FY  1993  field  test  validates  the  feasibility  and  usefulness  of 
RUMS,  the  Department  of  Defense  will  apply  RUMS  in  FY  1994.  However, 
the  field  test  showed  that  RUMS  as  then  constituted  was  not  useful. 

The  Director  and  the  Military  Departments  also  stated  that  the  purpose  of 
RUMS  was  never  intended  to  make  comparisons  between  test  and  evaluation 
facilities.  The  GAO  report  accurately  stated  the  purpose  as  one  of  a  number  of 
tools  facility  managers  have  to  measure  utilization  trends  within  their  activities. 

Audit  Response.  The  House  of  Representatives,  Committee  on  Appropriations 
Report  on  the  DoD  Appropriations  Bill  for  FY  1991  directed  that  the  Secretary 
of  Defense  institute  the  type  of  system  recommended  by  GAO  or  an  equivalent 
framework  by  June  1991  and  report  the  results  to  the  Appropriations 
Committees.  In  response  to  this  direction,  the  Department  issued  the  "Range 
Utilization  Measurement  Study  -  Field  Trial  Report"  July  1991.  This  field  trial 
report  indicated  to  Congress  that  a  range  utilization  system  would  be 
implemented  in  response  to  congressional  request.  However,  we  agree  with 
management  comments  that  RUMS  would  have  been  implemented  in  FY  1994 
if  the  field  tests  validated  the  feasibility  and  usefulness  of  the  system.  The 
appropriate  clarification  has  been  made  to  the  report. 

We  agree  that  activity  managers  can  use  RUMS  as  a  management  tool  to 
measure  utilization.  However,  the  intent  of  RUMS  was  more  than  to  provide 
local  managers  a  system  to  determine  testing  resource  utilization.  The  GAO 
report  states,  "The  Office  of  the  Secretary  of  Defense  (OSD)  is  responsible  for 
centralized  management  of  the  ranges  and  for  making  budgetary  decisions  on 
range  expansion,  consolidation,  and  improvements.  However,  the  ranges  do 
not  collect  and  report  standard  information  to  assess  overall  range  capacity  and 
use."  The  GAO  report  continues,  "our  proposed  framework  .  .  .  should  be 
useful  to  both  defense  and  congressional  decisionmakers."  The  GAO  report 
adds  that  "although  DoD  has  long  recognized  the  need  for  a  common 
measurement  of  its  test  ranges’  capacity  and  use,  the  ranges  do  not  collect 
comparable  data.  As  a  result,  DoD  cannot  readily  identify  either  excess  testing 
capacity  or  the  need  for  additional  capacity." 

The  intent  of  the  Appropriations  Committees  was  to  establish  a  utilization 
measurement  system  that  Congress  and  DoD  management  officials,  as  well  as 
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local  officials,  can  use  in  assessing  range  expansion,  consolidation,  and 
improvement  proposals.  We  agree  that  other  factors  must  be  considered  besides 
resource  utilization  in  evaluating  such  matters. 

Management  Comments.  The  Director  stated  that  audit  was  done  in  the  field 
without  those  providing  raw  RUMS  data  knowing  for  what  real  purpose.  The 
Director  stated  that  this  audit  was  done  during  the  period  in  FY  1994  when 
RUMS  was  in  transition  from  one  set  of  definitions  to  another  so  it  is  not 
surprising  that  inconsistences  resulted.  The  IG  auditors  misstated  the  purpose  of 
RUMS;  its  purpose  was  not  to  support  the  Base  Realignment  and  Closure 
Commission.  One  of  the  ground  rules  for  developing  the  system  was  the 
recognition  that  the  system  would  be  a  top-level  management  tool.  The 
Director's  comments  continued  to  stated  that  the  data  generated  by  this  system 
should  be  useful  in  trend  analysis  for  each  test  center.  The  Director's  comments 
stated  that  RUMS  should  show  year  to  year  utilization  profiles  and  be  useful  as 
a  flag  or  possible  further  analysis. 

Audit  Response.  We  agree  with  the  Director's  comments  that  the  Military 
Departments  recognized  problems  with  the  initial  set  of  RUMS  definitions.  The 
RUMS  Tri-Service  Ad  Hoc  Committee  developed  new  definitions  in 
August  1993.  As  identified  in  the  Military  Department's  comments,  the  Tri- 
Service  Ad  Hoc  Committee  decided  to  use  the  new  definitions  at  the  five  trial 
ranges  subject  to  approval  by  the  Board  of  Operating  Directors.  We  examined 
RUMS  data  subsequent  to  the  development  of  the  new  definitions  and  identified 
inconsistencies  in  the  definitions  usage. 

In  reference  to  the  use  of  RUMS  data  during  the  Base  Realignment  and  Closure 
Commission,  the  audit  report  stated  that  if  a  viable  system  existed,  the 
Commission  could  have  used  the  information,  with  other  data,  during  its 
deliberations.  We  agree  that  RUMS  is  a  tool  that  management  could  use  in 
evaluating  test  resource  usage  but  should  not  be  the  basis  of  any  decision.  Our 
reference  to  the  Base  Realignment  and  Closure  Commission  was  that,  if  the 
system  existed,  it  could  have  been  another  tool  for  the  Commission's  use. 

Management  Comments.  The  Director  and  the  Military  Departments 
comments  stated  that  White  Sands  Missile  Range  (WSMR)  may  not  have  had 
the  revised  guidance  during  the  visit  because  the  Board  of  Operating  Directors 
approved  the  guidance  in  September  1994.  The  Director  and  Military 
Department  continued  that  WSMR  and  other  test  centers  were  informed  of  the 
ad  hoc  committee's  recommendations  pending  final  approval.  The  comments 
added  that  the  basic  data  submitted  by  WSMR  to  Army  Test  and  Evaluation 
Command  (TECOM)  Headquarters  was  sufficient  to  calculate  both  the  old  and 
revised  parameters  and  that  these  calculations  were  done  at  TECOM 
Headquarters. 

Audit  Response.  The  new  definitions  were  developed  by  the  RUMS  Tri- 
Service  Ad  Hoc  Committee  in  August  1993  and  the  Ad  Hoc  Committee  agreed 
that  the  new  definitions  were  to  be  used  subject  to  approval  by  the  Board  of 
Operating  Directors.  Our  examination  was  conducted  during  the  summer  of 
1994,  providing  the  Army  ample  time  to  disseminate  the  new  definitions  to 
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WSMR.  In  reference  to  the  calculations,  the  RUMS  data  we  obtained  for 
WSMR  came  from  TECOM  and  not  from  WSMR.  As  noted  in  our  report, 
many  data  calculations  were  not  determined. 

Management  Comments.  The  Director  stated  that  unconstrained  capacity  for 
Type  B  resources  is  defined  as  16  hours  per  day,  250  days  per  year,  as  the  Air 
Force  Flight  Center  reported.  While  it  might  be  possible  to  operate  7  days  a 
week,  this  amount  of  operation  would  not  be  feasible  or  practical  for  extended 
periods  during  peacetime. 

Audit  Response.  We  agree  that  the  Air  Force  Flight  Center  is  a  TypeB 
resource;  therefore,  its  utilization  is  limited  to  16  hours  per  day  due  to  r^uired 
maintenance  and  test  preparation.  However,  under  the  definition  of 
unconstrained  capacity  (Appendix  A),  the  measurement  of  available  time  is 
determined  with  unconstrained  personnel  conditions.  Therefore,  we  do  not 
agree  that  this  resource  should  be  limited  to  5  days  per  week  as  opposed  to 
7  days  per  week. 

Management  Conunents.  The  Director  stated  that  the  audit  report  insinuates 
that  100  percent  of  test  facilities  should  be  reported.  He  stated  that  the  value  of 
doing  so  is  not  worth  the  additional  cost.  The  Director  said  the  nature  of  testing 
is  that  not  all  of  a  range's  capabilities  are  in  use  at  any  one  time.  One  can  view 
any  test  range  installation  as  a  tool  box  for  testing  and  many  factors  other  than 
utilization  alone  must  be  considered  to  ascertain  the  cost-effectiveness  of  these 
facilities. 

Audit  Response.  We  found  no  guidance  during  the  RUMS  audit  that  identified 
that  all  major  test  resources  are  not  to  be  reported.  The  test  resources  we 
reported  were  significant  resources  that  should  have  visibility.  We  agr^  that 
some  resources  may  have  a  low  utilization  because  they  are  unique  items. 
However,  RUMS  may  identify  other  test  resources  that  have  low  utilization  that 
justify  further  management  evaluation  to  determine  whether  the  test  resource  is 
still  needed.  Requiring  the  reporting  of  all  test  resources  denies  local  managers 
the  opportunity  for  selective  reporting. 
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Recommendation 

Reference 


1. 


2. 


3. 


Summary  of  Potential  Benefits 
Resulting  From  Audit 


Amount  and/or 

Description  of  Benefit  Type  of  Benefit 


Economy  and  Efficiency.  Requires  Nonmonetary, 

the  close  monitoring  of  the 
establishment  of  the  resource 
utilization  system. 

Economy  and  Efficiency.  Nonmonetary. 

Incorporates  a  specific  timeframe  to 

establish  a  utilization  measurement 

system  at  the  Major  Range  and  Test 

Facility  Base. 

Internal  Controls.  Establishes  Nonmonetary, 

appropriate  internal  controls. 
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Appendix  F.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Deputy  Director,  Test  Facility  Resources,  Office  of  Test  Systems  Engineering,  and 
Evaluation,  Washington,  DC 


Department  of  the  Army 

Headquarters,  U.S.  Army  Test  and  Evaluation  Command,  Aberdeen,  MD 

White  Sands  Missile  Range,  Las  Cruces,  NM 

U.S.  Army  Combat  Support  Activity,  Aberdeen,  MD 


Department  of  the  Navy 

Naval  Air  Systems  Command,  Arlington,  VA 

Naval  Air  Warfare  Center,  Arlington,  VA 

Naval  Air  Warfare  Center  -  Aircraft  Division,  Patuxent,  MD 


Department  of  the  Air  Force 

Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
Air  Force  Development  Test  Center,  Eglin  Air  Force  Base,  FL 
Air  Force  Flight  Test  Center,  Edwards  Air  Force  Base,  CA 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Under  Secfetary  of  Defense  (Comptroller/Management) 

Deputy  Under  Secretary  of  Defense  (Comptroller/Program/Budget) 
Deputy  Assistant  Secretary  of  Defense  for  Economic  Reinvestment  and  Base 
Realignment  and  Closure 

Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 

Director,  Test,  Systems  Engineering,  and  Evaluation 
Deputy  Director,  Test  Facilities  and  Resources 
Director  of  Defense  Research  and  Engineering 
Board  of  Operating  Directors  for  Test  and  Evaluation 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 
U.S.  Army  Test  and  Evaluation  Command 
Army  White  Sands  Missile  Range 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Comptroller  of  the  Navy 

Auditor  General,  Department  of  the  Navy 

Naval  Air  Systems  Command 

Naval  Air  Warfare  Center  -  Aircraft  Division 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Auditor  General,  Department  of  the  Air  Force 

Air  Force  Materiel  Command 

Air  Force  Development  Test  Center 

Air  Force  Flight  Test  Center 
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Defense  Agencies 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 
Inspector  General,  National  Security  Agency 
Inspector  General,  Central  Imagery  Office 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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Part  IV  -  Management  Comments 


Test  Systems  Engineering,  and  Evaluation 
Comments 


final  Report 

Referwf  _ ; _ 

OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

SOOO  DEFENSE  PENTAGON  ■f  6 

WASHINGTON  DC  S030I-9000  01  " 


MAR  ?  0 1::! 

MEMORANDUM  FOR  DIRECTOR,  ACQUISITION  MANAGEMENT 
DIRECTORATE,  Dod(IG) 

SUBJECT:  Draft  Audit  Report  "DoD  Range  Utilization  Measurement 
System" 


We  have  reviewed  the  subject  draft  audit  on  the  Resource 
Utilization  Measurement  System  (RUMS)  report  (Project  No.  4AB- 
5019.02),  January  18,  1995,  and  the  Service  comments.  Our 
specific  positions  and  comments,  along  with  those  of  the  three 
Services  are  attached.  We  endorse  the  Services'  comments.  Our 
non-concurrences  and  partial  concurrences  J^ave  generally  resulted 
from  two  primary  issues  with  the  audit:  conduct  of  the  audit  was 
premature,  and  a  misconception  of  the  auditors  as  to  the  purpose 
and  use  of  RUMS. 

First,  your  report  states  that  this  audit  was  requested  by 
my  office.  While  RUMS  would  be  an  appropriate  audit  subject 
after  its  implementation,  we  did  not  request  an  audit  of  RUMS 
during  the  period  when  it  was  in  development  and  transition:  we 
would  have  requested  it  after  FYSS,  when  a  final  RUMS  product  is 
to  be  available. 

Second,  the  purpose  of  RUMS  was  never  intended  to  make 
comparisons  between  TkE  facilities,  which  is  the  premise  of  your 
audit.  The  GAO  audit  which  caused  the  development  of  RUMS 
accurately  stated  its  purpose  as  one  of  a  number  of  tools 
facility  managers  have  to  measure  utilization  trends  within  their 
activities. 

As  a  result,  there  are  a  number  of  phraeings  and 
misstatements  in  the  draft  audit  report  that  resulted  from  a 
misunderstanding  of  the  background,  the  purpose  and  the  nature  of 
RUMS.  Consultation  with  my  staff  could  have  prevented  the  audit 
from  proceeding  to  the  degree  it  has.  While  our  detailed 
comments  on  the  draft  audit  findings  and  recommendations  are 
attached,  we  suggest  putting  this  audit  on  hold  and 
reaccomplishing  it  after  RUMS  has  been  fully  implemented. 


ACOUISmOM  AMD 
TicmeLPDY 
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Redirected 

and 

renumbered 

to 

Recommenda¬ 
tion  3. 


Deleted 


X>T8t4l  RR8PONS18  TO  RIC0KKSMDATI0M8  AlID  FZNDXK08 
DRAirr  XUDIT  RIPORT  OH  RAHOI  UTILItXTIOM  HZXSURRKIHT  6Y6TPI 
(PROJICT  4AB-S01f.0a) 

CooB&ontB  on  Rocooaondatloan: 


■1.  We  recommend  that  the  Director  for  Test,  Systems  Engineering 
and  Evaluation: 

■a.  Take  an  active  role  in  monitoring  the  implementation  of 
the  Range  [sic]  Utilization  Measurement  System  at  the  Major  Range 
and  Test  Facility  Base  to  ensure  its  accuracy  and  completeness.** 

Do  Mot  Concur,  since  3  991,  when  RUMS  was  first  conceived,  the 
Test  and  Evaluation  (T&E)  community  has  undergone  a  radical 
change  in  how  it. performs  functional  oversight.  Under  Project 
Reliance  there  now  is  a  Board  of  Directors  (BoD)  consisting  of 
the  service  vice  chiefs  of  staff.  They,  through  the  Board  of 
Operating  Directors  (BoOD)  oversee  the  joint  boards  that  for  each 
TtE  function  look  across  ali  Dod  to  evaluate  TtE  facility 
capabilities  and  shortfalls  and  recommend  changes  as  necessary. 
This  participatory  management  by  service  staffs  and  boards  of  the 
RUMS  initiative  may  have  caused  some  diminution  of  oversight  at 
OSD  of  RUMS  particulars.  However  the  Director  for  Test,  Systems 
Engineering  and  Evaluation  (DTSE&E)  is  briefed  frequently  in  his 
position  on  the  Defense  TtE  Steering  Group  and  is  satisfied  by 
RUMS  progress  to  date.  More  important  is  the  intense  oversight 
of  T&E  functions  throughout  Dod  provided  by  the  joint  BoD,  BoOD 
and  other  Reliance  boards. 

"b.  Establish  the  Range  [sic]  Utilization  Measurement 
System  as  an  assessable  unit  to  ensure  its  establishment  at  the 
Major  Range  and  Test  Facility  Base,  in  accordance  with  the  Dod 
Internal  Management  Control  Program,* 

Pytlally  Concur-  The  BoD  through  the  Board  of  Operating 
Directors  (BoOD)  already  has  taslced  the  services  to  innlement 
RUMS  by  the  end  of  FY9S.  The  DTSEtEtE  .fill  recommend  to  the  BoOD 
that  RUMS  be  designated  an  assessable  unit. 

c.  Report  in  the  Annual  Statement  of  Assurance  that  the 
lac)c  of  establishing  the  Range  (sic)  Utilization  Measurement 
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System  is  a  material  internal  control  weakness.* 

Dft  Wftt:  Concur.  RUMS  is  on  track  with  its  revised  schedule.  (Sec 
service  comments.) 

■2.  We  recommend  that  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  direct  the  implementation  of  a  Range 
[sic]  Utilization  Measurement  System  at  all  test  ranges  by  the 
end  of  FY95. 

Partially  Concur.  As  stated  under  l.b.  above,  the  BoD  through 
the  BoOD  already  has  tasked  the  services  to  implement  RUMS  by  the 
end  of  Fy95.  There  is  no  need  for  the  USD(AaT)  to  do  this. 

Genoral  Comments ; 

General  comments  are  keyed  to  page  numbers  and  paragraphs  in 
the  draft  audit. 

Exec  ■  Summary ^Introduct  ion.  .  And.  Finding...  Daq.e_  .6 . . last  Paragraph 

The  Office  of  the  Secretary  of  Defense  did  not  report  to 
Congress  that  implementation  at  all  test  ranges  would  be 
accomplished  by  the  end  of  FY94 .  The  Director  of  Defense 
Research  and  Engineering  did  state  in  a  9  November  1990  letter  to 
the  GAO’,  "If  analysis  of  the  field  test  conducted  during  Fy93 
validates  the  feasibility  and  usefulness  of  the  system,  the 
Department  will  apply  (RUMS]  to  appropriate  Major  Range  and  Test 
Facility  Base  activities  in  Fy94."  The  field  test  showed  that 
RUMS  as  then  constituted  was  not  useful.  The  Joint  Commanders 
Group  for  T&E  formed  an  ad  hoc  committee  which  developed  a 
standard  set  of  definitions  and  measurement  parameters  which  will 
be  implemented  throughout  the  MRTFB  by  the  end  of  Fy95. 

Exec .  Smiimary-Pb,jfigtlY.fiR 

Report  states  that  this  audit  was  part  of  the  audit  of 
■Expanded  Uses  for  the  Major  Range  and  Test  Facility  Base"  that 
the  Deputy  Director  for  Test  Facilities  and  Resources  requested. 
This  is  not  correct  as  an  audit  of  RUMS  would  not  be  requested 
during  the  period  when  RUMS  implementation  was  in  transition. 


*  Tri-Service  Resource  Utilization  Measurement  Study,  Field  Trial  Report^  July  1991 ,  page  A-1, 
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The  audit  was  done  in  the  field  without  those  providing  raw 
RUMS  data  knowing  for  what  real  purpose  they  were  doing  so.  This 
was  during  the  period  in  rY94  while  RUMS  was  in  transition  from 
one  set  of  definitions  to  another,  so  it  is  not  surprising  that 
there  were  inconsistencies. 

The  IG  auditors  have  misstated  the  purpose  of  RUMS.  The 
audit  would  have  one  believe  that  RUMS  has  failed  since  its 
purpose  was  to  support  the  Base  Closure  and  Realignment 
Commission  (BRAC)  and  it  didn't.  Not  only  was  that  not  its 
purpose  but  the  initial  study  responding  to  the  Congress 
specifically  stated  the  purpose  of  RUMS  to  be  internal 
management : 

"One  of  the  ground  rules  for  developing  the  system  was  the 
recognition  that  the  system  would  be  a  top-level  management 
tool.  The  data  generated  by  this  system  should  be  useful  in 
trend  analysis  for  each  test  center.  It  should  show  year  to 
year  utilization  profiles  and  be  useful  as  a  flag  for 
possible  further  analysis.  It  will  be  a  gross  indicator. , , 
[Emphasis  added] "  ^ 


"The  proposed  framework  contains  many  compromises  to 
accommodate  differences  in  missions  and  operating  practices 
among  the  MRTFB  installations.  We  believe  this  approach  may 
be  most  useful  for  measuring  an  activity  against  itself  by 
assessing  trends  based  on  past  performance  and  future 
projections...  [Emphasis  added]"  ^ 

Thus  RUMS  was  to  be,  and  is  now,  an  internal  utilization 
measurement  tool  never  meant  for  uses  such  as  BRAC.  It  was 
considered  for  use  during  BRAC  95,  and  determined  not  to  be 
useful  for  this  purpose  in  its  state  of  development. 


Test  Systems  EngioeerinSt  and  Eyaluation  CemmeDts 


Ffaud  Report 


The  criticism  of  insufficient  internal  controls  caused  by 
inadequate  monitoring  appears  to  be  premature,  given  that 
the  audit  occurred  during  a  transitional  time.  This 
transition  was  made  necessary  because  monitoring  ula 
effective. 


The  White  Sands  Missile  Range  is  not  at  Las  Cruces. 


The  Board  of  Operating  Directors  (BoOD)  is  not  under  the 
authority  nor  in  the  chain  of  coiisnand  of  the  Director  for 
Test,  Systems  Engineering  and  Evaluation. 


The  Army  states  that  although  the  individuals  contacted  by 
the  IG  team  ‘at  WSMR  may  not  have  had  the  revised  guidance 
during  the  visit,  that  was  during  the  transition  time  and 
before  the  revised  guidance  was  approved  by  the  Board  of 
Operating  Directors  in  September  1994.  "However,  WSMR  and 
all  other  test  centers  were  informed  of  the  ad  hoc 
committee's  recommendation  pending  final  BoOD  approval  and 
implementation.  The  basic  data  submitted  by  WSMR  to  [Amy] 
TtE  Command  (TECOM)  Headquarters  was  sufficient  to  calculate 
both  the  old  and  revised  parameters.  These  calculations  are 
done  at  TECOM  Headquarters." 


Page  6 


Unconstrained  capacity  for  Type  B  resources  is  defined  as  16 
hours  per  day,  250  days  per  year,  as  the  AF  Flight  Test 
Center  reported.  While  it  might  be  possible  to  operate 
seven  days  a  week,  this  %#ould  not  be  feasible  nor  practical 
for  extended  periods  in  peacetime. 


The  draft  audit  report  insinuates  that  100%  of  test 
facilities  should  be  reported.  The  value  of  doing  so  is  not 
worth  the  additional  cost.  Moreover  the  nature  of  testing 
is  that  not  all  of  a  range's  capabilities  are  in  use  at  any 
one  time.  This  often  allows  government  and/or  contractor 
employees  to  cover  several  different  test  facilities  that 


Page  9 
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are  in  uae  at  different  timea.  For  example,  aasurae  that 
test  cell  Z  is  one-of-a-kind,  is  used  only  once  per  year, 
and  then  is  ■covered*  by  employees  normally  working  test 
cell  A.  What  useful  information  is  imparted  by  reporting 
the  utilization  of  test  cell  Z  as  0,3%  for  the  year?  As 
stated  earlier,  the  purpose  of  RUMS  is  to  allow  facility 
managers  to  evaluate  utilization  trends  of  their  major  test 
properties,  in  conjunction  with  other  usage  parameters. 

One  can  view  any  MRTFB  installation  as  a  tool  box  for 
testing.  While  only  a  few  tools  may  be  in  use  at  any  one 
time,  it  is  always  handy  and  it  is  usually  efficient  to  have 
others  on  reserve  for  use  at  appropriate  times.  Similarly  a 
test  installation  has  facilities,  perhaps  on  reserve,  that 
are  essential  if  test  suj^rt  is  to  be  provided  when  needed. 
Many  factors  other  than  utilization  alone  must  be  considered 
to  ascertain  the  cost-effectiveness  of  these  faci^ties. 

■The  RUMS  should  be  used  in  conjunction  with  other  measures 
to  provide  a  broader  and  more  complete  assessment  of  MRTFB 
use.  Other  measures  could  consist  of  flight  hours,  labor, 
and  direct /indirect  funding. , -  Use  of  this  group  of 
measurements  would  reduce  misinterpretations  of  any  single 
measure .  ** 


The  services'  responses,  also  attached,  detail  the  reasons  why 
certain  facilities  were  not  included  as  "major"  facilities  for 
reporting. 
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HEHORWOUH  FOR  ITB  How»r4  «.  iMf,  IISAF  (R*t),  TIE 

Ex*cut!v»  AgcAt  loird  or  Olrtctor*,  AFAE,  U50  Rlr  Forci 


Ex*cut!v»  Agc«t  loird  of  Olroctoro,  AF/TE,  1650  Air  Foret 
Ftntogon,  Hiiklngton,  O.C.  20330-1650 


:  BoorO  of  Oportting  Otrocters  (BoOO) 
tnspoctor  Gonoril  (00016)  Ortft  Audit 
tion  Koosuronont  Sjrstoi  (RUNS)  (Frojic 


SUBJECT:  BoorO 
Oefanso  tnspocto 
Utilization  Hoas 
January  16,  199S 


Rasponse  to  Departaont  of 
Raport  on  "DoD  Rangt 


t  Audit  Raport  on  "DoD  Rangt 
(Frojact  4AB-S0l9.0t),"  Oatad 


The  BoOD  has  ravlawad  the  subject  DoDK  draft  audit  and  subalts  the  consents 
at  Enclosure  1  for  use  In  your  response  to  OSO.  Point  of  contact  for  this 
action  is  Nr,  Thonas  Nett,  DSN  f64-S0J3,  extension  3245. 


RICHARD  M.  TRAUMA 
Najor  Ctnera^^SA 
Chalman  u 


Nr.  Walter  W.  Hollis,  Deputy  Under  Secretary  of  the  Any  (Operations 
Research},  103  Amv  Pentagon,  Washington,  O.C.  20310-0103 
RADN  T.  D.  Ryan,  Chief  of  Naval  Oparatlons  (N091),  2000  Navy  Pentagon, 
Washington  D.C.  20350-3000 

RADH  William  E.  Netaian,  Coieiandtr,  Naval  Air  Warfare  Center,  1421  Jefferson 
Davis  Highway.  Suite  1400,  Ar1 Inton,  VA  22243-5000 
NG  Francis  C.  Gideon,  Jr.,  Oiractor,  Operations,  HQ  AFNC/DO,  4225  Logistics 
Avenue,  Suite  2,  Vrlght-Pattarsen  AFI,  OH  45433-5714 
Coenandar,  U.  $.  Anay  Ttst  and  Evaluation  Cemand,  ATTN:  ANSTE-TA  (Mr.  Gary 
Hollovay),  Aberdeen  Proving  Ground.  NO  21005-5055 
Coanander,  Naval  Air  Warfare  Center  Heapens  Division,  ATTN:  4KOOOOE/P03C  (Nr 
George  Smith),  Point  Nugu,  CA  93042-5000 
AFDTC/CD,  ATTN:  Or.  D.  Stewart,  Suite  117,  101  H.  D  Avenue.  Eglin  AFB,  FL 
32542-5495 

Hr.  Parker  Horner,  Chief,  Air  Force  Test  and  Evaluation  Resources  (AF/TER), 
1650  Air  Force  Pentagon,  Washington,  O.C.  20330-1650 
Dr.  John  Feulkes.  Chief,  Policy  Olvlslen,  U.S.  Army  Test  and  Evaluation 


1650  Air  Force  Pentagon,  Washington,  O.C.  20330-1650 
Dr.  John  Feulkes.  Chief,  Policy  Olvlslen,  U.S.  Anay  Test  and  Evaluation 
Management  Agency  (OACS-TE).  200  ArRjr  Pentagon,  Washington  O.C.  20310-0200 
CAPT  Elmer  Halley,  Director  of  Navy  Test  and  Evaluation  (N912).  2000  Navy 


Pentagon,  Washington,  O.C.  20350-2000 
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Execotly^  Stunmary 

Tbc  foUowing  is  a  summary  of  significant  items  and  issues  found  through  a  detailed  review 
of  the  Draft  Audit  Rq>oit  (4AB5019.02) 

The  Draft  Audit  Re|^  stares  that  DoD  had  not  complied  with  Coogressional  diitctioo  to 
implement  a  utilization  reporting  system  for  test  ranges.  Findings  of  the  audit  axe 
essentially  correct  as  to  the  Resource  Utilization  Measurement  System  ORUMS)  status 
within  DoD  at  the  time  of  the  audit  However,  the  nemise  of  audit  and  many  of  Ae 
reported  details  are  inconeci  or  misleading  <hie  to  AM  in  which  the  audit  was 
conducted. 

The  audit  was  conducted  during  the  second  half  of  FY  1994  while  RUMS  implezrientation 
was  still  in  progress.  The  conunitment  made  by  DoD  was  to  have  a  system  in>place  by  the 
end  of  FY  1994.  A  finished  RUMS  should  not  have  been  expected  at  the  time  of  the  audit 

The  Draft  Audit  Report  implies  that  Rl^MS  should  be  the  DoD  too!  to  allow  high  level 
management  decisions  on  the  future  of  it’s  Test  and  Evaluadon  (T&E)  facilities.  A  RUMS 
that  would  allow*  this  type  of  decision  making  would  be  prohibitively  expensive  since  it 
would  have  to  somehow  nonnalize  the  different  capabilities  and  management 
methodologies  existing  at  tbe  T&E  facilities.  RU^  is  a  commcm  tocM  that  can  be  used  by 
local  managers  to  manage  in  their  unique  environment.  It  should  not  be  used  to  make  high 
level  decisions. 

The  Draft  Audit  Report  continually  uses  RUMS  as  Range  Utilization  Measurement  System, 
the  acronym  now  stands  for  Resource  Utilization  Measurement  System. 
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TO  SPECmC  EECOMMENDATION8  COKTAINED  IN 
DRAFT  AUDIT  REPORT  ON  DOD  RANGE  UnUZATION 
MEASUREMENT  SYSTEM  (PROJECT 


FkallUpprt 


RECOMMENDATION  that  tbe  Director  Ibr  7^ 

Eoppeerm^  end  Evihation  take  in  active  mV  in  tMtMtnriny 

Rac^  UtiluatiocMeasureiDeiit  System  « the  M^Raafe  and  TSni^i^Base  10  easm 

its  accuracy  and  completencu." 

Part  ally  CoDCyt  Tbc  Director  Test,  Systems  Ezigineerizii  lad  EvahiatioD  as  ibe 
Major  apd  Tesi  Facility  Baiie  (MRTFB)  oversJ^t  aiitliority  las  always  had  an  active 
foIc  w  tbc  moQitonni  of  RUmS  impktneataiioo-  Pcnodic  bnefiz^  ktve  been  rn^  to  him 
wkJ  hiS  $1^00  the  progress  aod  status  of  RUMS  iJBq)IciDentatj^ll^  Board  of  Operating 
EXwtors  (B<^D)  is  charged  with  tbc  implcmtmatioo  and  reponiMcf  RUMS  to  the  Board 
of  Directon  (BoD).  • 

RECQ^^^IENDATIQN  1J>i  *y^t  recommend  that  the  Director  Ibr  Test,  Systems 
Engineering  tod  Eviluarioo  cstablisb  d)e  Range  Utilizatioo  Mcasuremcot  System  as  an 
assessibJc  unit  lo  ensure  its  establishment  at  &  Major  Range  and  Test  Fa^ty  Base  m 
accordance  with  the  DoD  Internal  Management  Cootrol  Program.” 

CyMt.  The  B^D  is imjdementiDf  RUMS  u  ill MKITB  ihes  durini  FY 
1 995.  BoOD  has  tasked  ibe  Semets  with  unplcmenting  and  gallaafeg  data  in  the  first 
quancr  FY  1995,  rcconciliaiion  of  dau  between  the  services  in  tbe  second  quarter  FY 
1995,  and  full  impJcmemaiion  of  a  common  RUMS  by  the  end  of  FY  1995. 

DoD  DiitcUyc  5010.38,  TnternaJ  Management  Control  Progmn,”  leqoires  Federal 
managers  to  iiopjcment  a  compreheDsive  system  of  internal  managamt  controls  and 
iden'Jfy  assessable  units  that  will  provide  reasonable  assurance  that  41^  ujcj  properlv 
and  orograms  are  cffecii  vely  and  efficiently  managed.  RUMS  will  become  iDOthcr  TOlfor 
^  iTMagcrs  to  efficiectl)  manage  their  facility  m  their  unique  ocentmi  environment 
RUMS  is  not  a  too)  that  will  comput  ooe  facili^*  with  another  for  micro  decision  making. 

RECOMMEyPATlQN  l,c.  ‘^erecomnicod  that  the  Director  Test.  Systems 
Ergineenng  and  Evaluatioo  report  in  tbe  Annual  Statement  of  Assuaoce  Chat  tte  lack  of 
establishing  the  Range  Utikunon  Measurement  System  is  a  material  imemaJ  cooool 
weakneas.** 

Do  ypt  CoPCUn  DDR&Emre5)oodkigtoiheGcncrilAccotmtmgOfficc(NSlAD90- 

9 1 )  laid  out  the  schedule  for  RUMS  process  dc velopmcni  as  follows: 

FY  1991  a  utiluarioo  measurement  system  would  be  dbsigted  and  debugged. 

FY  1992  tbe  system  would  be  field  tested  at  several  MRTFB  privities. 

FY  1993  the  usefulness  and  feasibility  of  the  lysim  wouiii 

FY  1994  tbc  system.  If  useable,  would  be  implemented  to  ipnopriate  MRTFB 
activities. 

In  Dumber  1992  represenutives  of  the  three  services  met  10  review  the  data  and  evaluate 
the  effcctrycncss  of  RUMS.  AH  attendees  agreed  that  RUMS  as  was  then  implemented  did 
not  meet  OSD,  Service  or  Md  level  requirements. 


Redirected  and 
renumbered  to 
Recommendation 

3. 


Deleted 
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As  I  result  of  the  shortcomings  of  the  original  RUMS,  the  Joint  Commanders  Groim  for 
TAE  (JCG{TA£)]  formed  an  ad  hoc  committee  to  mo^  RUMS  into  something  which 
would  be  useful.  This  ad  boc  committee  has  developed  a  standard  set  of  deflnltioitt  and 
measurement  parameters  which  are  now  being  deconflicted  and  wih  be  hxmlemenied  across 
the  MRTFB  by  the  end  of  FY  1995. 

RECOMMENDATION  2.  •*^’e  recomn^nd  that  the  Under  Secretary  of  Defense  for 
Acquisition  direct  the  implementation  of  a  Range  Udlizatioo  Measurement  System  ar  all  test 
ranges  by  the  end  of  FY  1995.** 

Do  Not  concur.  The  BoOD.  as  agents  of  the  Board  of  Directors  (Service  Executive 
Agenu  responsible  for  oversight  of  the  T&E  Infmtructure),  has  already  directed  that  the 
RUMS  be  fully  implemented  across  the  MRTFB  by  the  end  of  FY  1995. 
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RESPONSES  TO  SPECIFIC  FINDINGS  CONTAINED  IN  MUFT  AUDIT 
REPORT  ON  DOD  RANGE  UTILIZATION  MEASUREMENT  SYSTEM 
(PROJECT  4AB«5019.02) 

FINDING.  (Page  6,  Pfera^nph  1,  Last  sentence)  **As  a  result,  Ibe  OSD  nd  tbe  Military 
Depamnents  cannot  make  accurate  decisioDS  consolidation,  expansk^  and 
improvemeots  at  the  test  ranges  based  on  credible  assessments  of  capaciy  and  ose." 

Do  not  Concur.  A  theme  is  woven  throughout  the  Draft  Audit  R^xxt  that  RUMS  will 
allow  high  level  policy  makers  to  make  fMniity  closure,  consolidatkn,  and  investment 
decisions.  This  is  not  true.  A  RUMS  to  achieve  that  goal  will  be  prohibitively  e^nsive 
since  it  will  have  to  somehow  normalize  the  various  range  capabilities  and  opoational 
methodologies.  RUMS  can  be  used  by  local  managers  to  assist  in  their  temty 
management.  RUMS  data  for  a  facility  is  unique  to  Chat  facility,  taking  into  account  the 
capability,  opcraiiooal  methodology,  test  con:q>lexitje5,  and  architecture  of  that  panicular 
facility.  Without  a  through  knowledge  of  what  RIMS  means  at  each  facility,  a  side-by- 
side  comparison  of  similar  facilities  using  only  RUMS  data  is  meaningless. 

FINDING.  (Page  7.  first  three  lines)  “...during  FY  1993,  DoD  would  assess  the 
feasibUity  and  usefubess  of  RUMS,  and  DoD  would  implement  RUMS  at  all  19  test  ranges 
by  the  end  of  FY  1994.” 

Do  not  concur.  RUMS  would  only  be  implemented  if  useful.  This  would  be  based 
upon  the  fcasibilitj  and  usefulness  assessment  of  RUMS  done  in  FY  1993.  In  December 
1 992  reprcscnlau^  cs  of  the  three  services  met  to  review  the  data  and  evaluate  the 
effectiveness  of  RIMS.  All  attendees  agreed  that  RUMS  as  was  then  m^lcmentcd  did  not 
meet  OSD,  Service  -or  field  level  requirements. 

As  a  result  of  the  shortcomings  of  the  original  RUMS,  the  Joint  Commanders  Group  for 
T&E  ;JCG(T&E)  formed  an  ad  hoc  committee  to  modify  RLMS  into  something  which 
would  be  useful.  This  ad  hoc  committee  has  developed  a  standard  set  of  definiuons  and 
mcasuremcm  parameters  which  art  now  being  dccooflicted  and  will  be  implemented  across 
the  MRTFB  by  the  end  of  FY  1995. 

FINDING.  (Page  8,  Definitions  and  Measurement  Parameters,  second  paragraph) 

po  not  concur.  The  auditors  during  their  visits  to  the  MRTFB  sites  did  not  identify 
themselves  as  auditing  RIMS  nor  did  they  inquire  directly  about  RUMS  mylemeptalion  or 
its  status.  They  stated  they  were  conducting  an  audit  of  expanded  use  of  the  MRTFB  for 
training  puiposes  and  requested  RUMS  dau  to  assist  in  this  effeti.  They  did  not  inquire 
directly  about  the  status  of  RUMS  implementaiioD.  They  did  not  inquire  about  preliminary 
facilin  identification  and  ciq»city  information  for  the  test  centers.  Tneir  specific  request 
was  only  for  data  in  RUMS  fonnaL  They  were  provided  with  only  relatively  raw  use  data 
on  the  designated  trial  facilities. 

White  Sands  Missile  Range  (WSMR)  was  cited  for  reporting  inconecl/ouidated  parameten 
because  they  had  rtceiveo  no  revised  guidance.  At  the  time  of  the  audit,  the  revued 
definitions/parameters  had  not  yet  been  approved  for  implementation  by  the  BoOD.  They 
were  subsequently  approved  by  the  BoOD  in  September  1994.  However,  W'SMR  and  all 
other  test  centers  were  infonned  of  the  ad  hoc  committee's  recommeodations  pending  final 
BoOD  approval  and  implementation.  The  basic  data  submitted  by  WSMR  to  T&E 
Command  (TECOM)  Headquarters  was  sufficient  to  calculate  the  old  axKl  revised 
parameters.  These  c^culations  are  done  at  TECOM  Headquarters. 


Fteal  Repart 

SfegCT, 


Pages  6  and  7 
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piNDlNG.  (Pafies  8  and  9.  Definitions  and  Measurement  Parameters,  third  panfrapb) 
•The  Air  Force  Eight  Test  Center  at  Edwmd  Air  Force  Base  was  not  reporting 
*^mconstraiDed  capacity**  in  accordance  with  applicable  guidelines.** 

pn  not  conciir.  The  Ak  Force  (kilitw  mcrtioned  in  ti)C  Draft  Audit  Report  ait  Type  B 
Resources  whose  unconstrained  capacily  is  defined  as  16  hours  per  day  for  250  days  per 
year.  The  rational  for  2^  days  per  year  instead  of  365  days  per  year  is  that  a  16  hour,  5 
day  work  week  is  normal  for  a  two  shift  cyeration.  The  facility  could  be  used  7  days  a 
week  using  overtime,  but  not  for  exteoded kngths  of  time  unless  under  emergency 
situations.  A  revision  in  the  definition  to  a  7  days  per  week  operation  could  be  done  and 
would  affect  only  unconstrained  c^iacity  and  surge  conoputanoos,  not  utilization  or  use 
efficiency  computations. 

FIN  DING.  (Page  9,  Methodology  and  Tcchniaues,  first  paragrafA)  “We  reviewed 
methodologies  and  techniques  for  gathering  RUMS  dau  at  four  of  the  five  field  trial  test 
nmges  (the  trial  ranges)  and  found  that  some  test  facilities  and  capabilities  were  not  reported 
and  reporting  was  inconsisicnt.’* 

Concur.  The  trial  ranges  were  continuing  to  gather  data  using  the  original  RUMS  which, 
in  December  1992.  representatives  of  the  three  services  agreed  did  not  meet  OSD.  Service 
or  field  level  requirements.  The  RUMS  being  implemented  at  the  present  time  was  change . 
in  part  to  remove  inconsistencies. 

FINDING.  (Page  9,  Methodology  and  Techniques.  Unreponed  Test  Facilities) 

Dn  not  concur.  The  Draft  Audit  Report  refers  frequently  to  a  Tri-Service  RUMS 
Commince  created  in  May  1993  and  implies  it  had  a  written  charter  to  issue  policy  and 
implement  a  sj  stem.  Tbc  auditors  are  rtfciiing  to  the  ad  hoc  group  established  by  the 
JCG(T&E)  in  March  1993.  This  ad  hoc  group  w  as  tasked  to  review'  prior  efforts,  devise 
alicmative  methodologies  and  parameters,  and  report  back  to  the  JCG(T&E).  They  were 
not  i^ed  TO  issue  pobey  and  implement  a  system. 

RUMS  is  not  intended  to  report  every  facility:  each  MRTFB  will  detenninc  its  major 
capabilities  and  report  on  them.  Specifically  the  report  cites  WSMR  for  failing  to  rqx»t  on 
fi\e  resources  that  ibc  auditors  deemed  to  be  reportable: 

The  Electronic  Warfare  Vulnerability  Facility  is  a  scientific  and  technical  facility 
owned  and  operated  by  the  Army  Research  Laboratory,  a  tenant  at  WSMR. 

Tbc  Hijh  Speed  Test  Track  is  a  HoDoman  Air  Force  Base  facility  that  was  not  one 
of  the  field  test  ranges. 

Tbc  Cbcmistiy  and  Metallurgy  Laboratory  ai  WSMR  is  a  supporting  facility,  not  a 
stand  alone  test  facility.  It  is  generally  used  to  do  post  icst/failure  tasks  such  as 
toxic  fumes  analysis,  oil  saxt^ile  analysis,  etc. 

The  Optics  Test  Laboratory  at  WSMR  belongs  to  its  Insmimentaiioo  Development 
Directorate  and  is  used  to  cabbratc,  repair,  or  develop  instrumentation.  This 
laboratoiy  docs  not  test  defense  hardware. 

The  Range  Control  Center  is  an  integral  part  of  the  WSMR  Test  Range  Complex. 

Its  utilization  is  the  same  as  the  Test  Range  Complex.  Range  and  misskm  control 
facilities  are,  by  definition,  included  as  pan  of  such  complexes  (Appendix  B, 
Resource  Utilization  Measurement  System  Functional  Categories.  Number  18). 
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piKDING.  (Pages  9  and  10,  Medtodology  nd Techniques,  Inconsistent  Reporting) 

Concur.  The  RL^  being  iixq»lemeoted  at  the  present  tune  was  changed  in  part  to 
remove  inconsistencies. 

yiNDING.  (Page  10,  Monitoring  Inq>]ementttioD  Effort,  first  paragraph) 

po  not  eoncur^Tl^  ad  iKxcoimmttee  was  tasked  to  revkwpriw  efforts,  devise 
alternative  methodologies  and  parameters,  and  itpdit  back  to  the  JCG(Td^).  It  was 
formed  to  address  issues  that  made  RUMS  (as  defined  at  that  time)  unusable,  some  of 
which  w  ere  inconsistencies  in  reporting. 

PINDING.  (Page  10,  Monitoring  Implementadoo  Effort,  second  and  third  paragnphs) 

Do  not  concur.  RUMS  is  not  intended  to  repwt  every  facility;  each  MRTFB  will 
determine  its  major  capabilities  and  leport  on  them. 

FINDING.  (Page  10,  Monitoring  Implcmenlatior  Effort,  fourth  paragraph) 

“...Reporting  less  than  100  percent  of  test  capabilities  and  usage  would  prohibit  OSD 
xnanagemcnl  from  comparing  all  testing  resources.” 

Do  not  foncur.  A  RUMS  to  allow  comparison  of  similar  capabilities  will  be 
prohibitiveh.  expensive  since  it  will  have  to  somehow  normaliac  the  various  range 
capabilities  and  operational  methodologies,  RUMS  can  be  used  by  local  managers  to  assist 
in  their  facility  omagcmcm.  RUMS  data  for  a  facility  is  unique  to  that  facility,  taking  into 
account  the  capability,  operational  imtbodology,  lest  complexities,  and  architecture  of  that 
particular  facDiiy.  Without  a  through  knowledge  of  what  RUMS  means  at  each  facility,  a 
side-by-side  comparison  of  similar  facilities  using  only  RUMS  data  is  meaningless. 

FINDING.  (Page  11.  Use  of  Utiliaation  Dart) 

Dn  not  concur.  A  RUMS  to  tDow  comparison  of  similar  capabilities  to  make  BRAC 
type  decisions  will  be  prohibitive!)*  expensive  since  it  will  have  to  somehow  normaliae  the 
various  range  capabilities  and  operational  methodologies.  RUMS  can  be  used  by  local 
managers  to  assist  in  their  facility  management.  RUMS  data  for  a  facility  is  unique  to  that 
facility,  taking  into  account  the  cimability,  operational  methodology,  lest  complexities,  and 
architecture  of  that  particular  faciliry.  Without  a  through  knowledge  of  what  RUMS  means 
at  each  facility,  a  side-by-side  com^son  of  similar  facilities  using  only  RUMS  dau  is 
meaningless. 

FINDING.  (Pages  11  and  12, Conclusion) 

Dn  no!  concur.  The  MRTFB’s,  under  direction  from  the  BoOD,  are  implejncnting 
RUMS  with  full  implcmentatioD  due  by  the  end  of  FY 1995. 
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MEMORANDUM  F{»  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 

FROM:  HQUSAF/TE 

1650  Air  Force  Pentagon 
Washington,  DC  20330-1650 

SUBJECT:  DoDIG  Draft  Report,  "DoD  Range  Utilization  Measurement  System," 

(Project  No.  4AB-5019.02) 

This  is  in  reply  to  your  memorandum  requesting  that  the  Department  of  Defense  provide 
comments  on  the  subject  report 

I  have  reviewed  the  draft  audit  on  the  Range  Utilizauon  Measurement  System  report  and 
can  only  partially  concur  with  your  recommendations.  I  believe  that  the  name  of  the  process  was 
changed  from  '*Kange  Utilization  Measurement  System”  to  **R<source  Utilization  Measureraem 
System  (RUMS)”,  and  suggest  the  title  of  the  report  be  corrected. 

I  concur  with  the  detailed  comments  provided  by  the  Board  of  Operating  Directors 
(BoOD),  and  the  Diieaor  for  Test.  Systems  ^gineering  and  Evaluation's  position  which  expands 
on  the  BoOD  comments.  My  comments  on  the  recommendations  for  corrective  action  are  as 
follows: 

Recommendation  l.a.  -  Do  Not  Concur.  Previous  efforts  lo  implement  the  RUMS  were 
accomplished  under  the  auspices  of  the  Joint  Commander  Group  for  T&£.  A  recent 
improvement,  the  Implementation  of  the  T&E  Tri-Service  Executive  Agent  structure,  has  now 
taken  effect  Recommend  that  management  of  the  RUMS  effort  be  retained  within  the  Board  of 
Operating  Directors  responsibility.  An  increased  DoD  role  monitoring  the  implementation  of 
RUMS  is  not  required  within  this  new  tri-services  structure.  Plans  and  status  should  continue  to 
be  provided  to  the  Director  or  Test,  Systems  Engineering  and  Evaluation. 

Recommendation  Lb.  -  Partially  Concur.  The  Board  of  Directors,  through  the  Board  of 
Operating  Directors,  is  already  providing  sufficient  management  and  oversight  for  RUMS. 

Recommendation  l.c.  •  Do  Not  Concur.  It  should  be  reported  that  progress  has  been 
made,  and  previous  managerial  deficiencies  within  the  Military  Services  regarding  oversight  of  the 
RUMS  effort  have  been  corrected  with  this  implemenution  of  the  Board  of  Operating  Directors. 
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Recommendation  2.  •  Partially  Concur.  The  Board  of  Directors,  within  the  TStE  Tri- 
Service  Executive  A^ent  structure,  have  directed  the  Board  of  Operacini  Directors  to  work  with 
the  Services  an  implement  RUMS  by  the  end  of  FY  95. 

1  would  emphasize  that  the  rinding...‘t)SD  and  the  Military  Departments  can  not  make 
accurate  decisions  on  consolidation,  expansion,  and  improvements  at  the  lest  ranges...**,  page  6.  is 
an  extreme  oversutement  of  the  value  of  RUMS.  Inier-^tia-Service  consolidation  efforts 
through  Reliance.  T&E  Executive  Agent,  and  BRAC  have  shown  that  a  czediUe  decision  making 
process  is  complex  and  multi-faceted.  A  valid  utilization  measurement  system  will  be  an 
important  part  of  that  process,  but  should  be  the  single,  primary  basis  for  decisions. 

4)HNT.  manclark 

Deputy  Director.  Test  &  Evaluation 
cc: 

DoD/DTSE&E 
AF/CV  (BoD  Chairman) 

SAF/FMP 

l)DTtE/TFC 
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